Gaussian Process

In probability theory and statistics, a Gaussian process (GP) is a stochastic process (a collection of random variables indexed by time or space), such that every finite collection of those random variables has a multivariate normal distribution, i.e. every finite linear combination of them is normally distributed. The distribution of a Gaussian process is the joint distribution of all those (infinitely many) random variables, and as such, it is a distribution over functions with a continuous domain, e.g. time or space. A machine-learning algorithm that involves a Gaussian process uses lazy learning and a measure of the similarity between points (the kernel function) to predict the value for an unseen point from training data. The prediction is not just an estimate for that point, but also has uncertainty information - it is a one-dimensional Gaussian distribution (which is the marginal distribution at that point).

Resources

• https://en.wikipedia.org/wiki/Gaussian_process
• http://scikit-learn.org/stable/modules/gaussian_process.html
• A Practical Guide to Gaussian Processes
• A Visual Exploration of Gaussian Processes
  • https://www.jgoertler.com/visual-exploration-gaussian-processes/
  • https://blog.dominodatalab.com/fitting-gaussian-process-models-python/
• Gaussian processes
• Deep Neural Networks and Gaussian Processes: Similarities, Differences, and Trade-Offs

Code

• #CODE GPy
• #CODE GPyTorch
  • https://gpytorch.ai/
• #CODE GPFlow
  • https://gpflow.readthedocs.io/en/master/intro.html
• #CODE GPflux
  • GPflux uses the mathematical building blocks from GPflow and marries these with the powerful layered deep learning API provided by Keras.
  • https://secondmind-labs.github.io/GPflux/tutorials.html
• #CODE Gpax - Structured Gaussian Processes and Deep Kernel Learning
  • GPax is a small Python package for physics-based Gaussian processes (GPs) built on top of NumPyro and JAX

Books

• #BOOK Bayesian Optimization Book (Garnett 2021)

References

• #PAPER Gaussian Processes for Machine Learning (Rasmussen and Williams 2006)
  • http://www.gaussianprocess.org/gpml/chapters/RW.pdf
• #PAPER Convolutional Gaussian Processes (van der Wilk 2017)
  • #CODE https://gpflow.readthedocs.io/en/master/notebooks/advanced/convolutional.html
• #PAPER Deep convolutional Gaussian processes (Blomqvist 2018)
  • #CODE https://github.com/kekeblom/DeepCGP
  • https://github.com/kekeblom/DeepCGP/blob/master/notebooks/Inspect.ipynb
• #PAPER Gaussian processes meet NeuralODEs: A Bayesian framework for learning the dynamics of partially observed systems from scarce and noisy data (Aziz Bhouri 2021)
  • #CODE https://github.com/PredictiveIntelligenceLab/GP-NODEs
• #PAPER #REVIEW An Intuitive Tutorial to Gaussian Processes Regression (Wang 2021)
• #PAPER #REVIEW Deep Gaussian Processes: A Survey (Jakkala 2021)
• #PAPER Physics makes the difference: Bayesian optimization and active learning via augmented Gaussian process (Ziatdinov 2021)